Efficient allocation of network resources, such as available network bandwidth, has become critical as enterprises increase reliance on distributed computing environments and wide area computer networks to accomplish critical tasks. The widely-used TCP/IP protocol suite, which implements the world-wide data communications network environment called the Internet and is employed in many local area networks, omits explicit supervisory function over the rate of data transport over the various devices that comprise the network. While there are certain perceived advantages, this characteristic has the consequence of juxtaposing very high-speed packets and very low-speed packets in potential conflict and produces certain inefficiencies. Certain loading conditions degrade performance of networked applications and can even cause instabilities which could lead to overloads that could stop data transfer temporarily. The above-identified U.S. patents and patent applications provide explanations of certain technical aspects of a packet based telecommunications network environment, such as Internet/Intranet technology based largely on the TCP/IP protocol suite, and describe the deployment of bandwidth management solutions to monitor and/or manage network environments using such protocols and technologies.
The management of such networks requires regular monitoring and collection of data characterizing various attributes of the network, its operation and/or the traffic flowing through it. For example, Cisco Systems, Inc. of San Jose, Calif. offers a feature set of data monitoring and collection technologies in connection with its routers, called Netflow®. The Cisco IOS® NetFlow feature set allows for the tracking of individual IP flows as they are received at a router or switching device. According to the technology, after a flow has terminated, a suitably configured router or switch generates a NetFlow record characterizing various attributes of the flow. The NetFlow record is ultimately transmitted as a datagram to a NetFlow Data Collector that stores and, optionally, filters the record. A NetFlow Record includes a variety of attributes, such as source and destination IP addresses, packet count, byte count, start and end time stamps, source and destination TCP/UDP ports, Quality of Service attributes, and routing-related information (e.g., nexthop and Autonomous System (AS) data). Such NetFlow® records are similar to call records, which are generated after the termination of telephone calls and used by the telephone industry as the basis of billing for long distance calls, for example.
Most network devices maintain data characterizing utilization, operation and/or performance of the network devices, and/or the network on which the devices operate, in limited, volatile memory, rather than using persistent storage (e.g., hard disks or other non-volatile memory). Consequently, network management applications commonly use the Simple Network Management Protocol (SNMP) to poll network devices (using the Management Information Base (MIB) associated with the network device) at regular time intervals and maintain the sampled raw data in a persistent data store. The network management application, such as a reporting package, then processes the raw data to allow for the creation of reports derived from the raw data detailing operation and/or performance of the device and/or the network. Management Information Bases typically contain low-level information characterizing the operation of the network device, such as the number of bytes or packets encountered on an interface, and do not provide information concerning the characteristics of data flows.
Using a reporting package, a network administrator may then analyze the data to yield information about the performance or utilization of the network and/or network devices associated with the network. Indeed, Various applications can then access the Data Collector to analyze the data for a variety of purposes, including accounting, billing, network planning, traffic engineering, and user or application monitoring. There are public-domain implementations of collectors for standard NetFlow records. These are, however, unable to answer questions such as “which hosts are running the busiest Kazaa (or other peer-to-peer file sharing) servers” (as NetFlow records are not suitable for analyzing and classifying network traffic that does not use registered IP port numbers).
Packeteer, Inc. of Cupertino, Calif. develops bandwidth monitoring, management, and reporting software and systems. Its PacketSeeker® systems and PacketShaper® bandwidth management devices, among other things, provide “application aware” monitoring of network traffic enabling classification of network traffic flows on a per application basis. The Packetshaper® bandwidth management device includes functionality allowing for classification of network traffic based on information from layers 2 to 7 of the OSI reference model. As discussed in the above-identified patents and patent applications, the bandwidth management device includes a measurement engine operative to record or maintain numeric totals of a particular measurement variable at periodic intervals on a traffic classification basis. The bandwidth management device further includes a management information base including standard network objects maintaining counts relating, for example, to the operation of its network interfaces and processors. Packeteer's ReportCenter™ leverages the powerful network utilization and application performance statistics available in Packetshaper® bandwidth management devices and offers a centralized reporting platform to monitor and manage large deployments efficiently by streamlining collection, collation, storage, analysis, and distribution of measured statistics.
While the measurement engine is sufficient to achieve its intended purpose, some useful data for analyzing network usage and/or diagnosing problems is not available historically, but is only kept in memory while the PacketSeeker, PacketShaper or other bandwidth management device is running. In particular, the reports on “top talkers” and “traffic history” are not available for specific intervals in the past nor available after the device crashes, possibly due to some kind of attack or power outage. Furthermore, data maintained by the measurement engine, is generally not flow-based, and cannot answer questions like “which clients are running port scanners.” Furthermore, as discussed above, NetFlow records characterize individual flows; however, standard NetFlow records cannot answer such questions or others requiring classification of flows beyond the attributes maintained by NetFlow records.
In light of the foregoing, a need in the art exists for methods, apparatuses and systems that enable a flow-based, traffic-classification-aware data collection and reporting system. A need further exists in the art for methods, apparatuses and systems allowing for enhanced informational queries relating to the operation of networks. Embodiments of the present invention substantially fulfill these needs.